Designing Power-Efficient Modulation Formats for 
Noncoherent Optical Systems 



Johnny Karout*, Erik Agrell*, Krzysztof Szczerba^^ and Magnus Karlsson^^ 
* Department of Signals and Systems 
1^ Department of Microtechnology and Nanocience 
Chalmers University of Technology, SE-412 96 Goteborg, Sweden 
Email : j ohnny.karout @ chalmers . se 



Abstract — We optimize modulation formats for the additive 
white Gaussian noise channel with a nonnegative input con- 
straint, also known as the intensity-modulated direct detection 
channel, with and without confining them to a lattice structure. 
Our optimization criteria are the average electrical and optical 
power. The nonnegativity input signal constraint is translated into 
a conical constraint in signal space, and modulation formats are 
designed by sphere packing inside this cone. Some remarkably 
dense packings are found, which yield more power-efficient 
modulation formats than previously known. For example, at a 
spectral efficiency of 1 bit/s/Hz, the obtained modulation format 
offers a 0.86 dB average electrical power gain and 0.43 dB 
average optical power gain over the previously best known 
modulation formats to achieve a symbol error rate of 10~^. 
This modulation turns out to have a lattice-based structure. At a 
spectral efficiency of 3/2 bits/s/Hz and to achieve a symbol error 
rate of 10~^, the modulation format obtained for optimizing the 
average electrical power offers a 0.58 dB average electrical power 
gain over the best lattice-based modulation and 2.55 dB gain 
over the best previously known format. However, the modulation 
format optimized for average optical power offers a 0.46 dB 
average optical power gain over the best lattice-based modulation 
and 1.35 dB gain over the best previously known format. 

I. Introduction 

Multilevel modulation has attracted significant research in- 
terest with its ability to improve the spectral efficiency of 
communication systems. The enabling technology behind it 
is the coherent transmission and detection, which gives access 
to both the carrier amplitude and phase to carry information. 
However, this increased spectral efficiency comes at the ex- 
pense of a reduced power efficiency, which is undesirable in 
systems where power consumption is a constraint. Therefore, 
designing modulation formats which offer a good trade-off 
between spectral and power efficiency becomes challenging. 
Using lattice codes, which are a finite set of points selected out 
of an A^-dimensional lattice, is one approach which has been 
extensively used in the construction of multilevel modulation 
formats for additive white Gaussian noise (AWGN) channels 
with coherent detection |[T|-|[3|. In addition, techniques such 
as constellation shaping and nonequiprobable signaling have 
been used to minimize the average power ||2|, ||4j. The former 
is done by selecting the set of points in a lattice which 
have minimum energies, whereas the latter minimizes the 
average power by reducing the transmission frequency of 
points with high energies. Another approach is by resorting 
to numerical optimization techniques to find the best possible 



packing of constellation points as in f5l-f9l for different power 
constraints, whether average or peak power. The drawback is 
the lack of geometric regularity, which increases the modulator 
and demodulator complexity. As the number of constellation 
points increases, the best known packings approach a regular 
structure such as a lattice | 3, Ch. 1]. 

Both approaches for designing power-efficient modulation 
formats assume that both the amplitude and phase of the carrier 
can be used to carry information. However, in systems where 
phase information is absent, different modulation techniques 
must be considered. Examples of such systems include phase 
noise limited systems, and noncoherent systems where infor- 
mation is encoded onto the amplitude of the carrier and the 
envelope of the received signal is detected at the receiver, etc. 
The latter is prevalent in optical communication systems where 
the overall cost and complexity is a critical constraint. Such 
type of noncoherent systems are known as intensity -modulated 
direct-detection (IM/DD) systems and will be the focus of our 
work. In such systems the information is encoded onto the 
intensity of the optical carrier, and this intensity must, at all 
time instances, be nonnegative. Applications using IM/DD are, 
for example, wireless optical communicatio ns |10| |-|12||, and 
short-haul fiber links including data centers |I13|. 

In the absence of optical amplification, IM/DD systems can 
be modeled as a conventional AWGN channel whose input is 
constrained to being nonnegative |10, Ch. 5], |11|, |14|-|17J. 
Since the optical phase cannot be used to carry information, re- 
sorting to multilevel pulse amplitude modulation (M-PAM) is 
a natural low-complexity way of extending the widely spread 
on-off keying (OOK) to improve spectral efficiency. However, 
this is different from the conventional PAM since no negative 
amplitudes can be used 1 10, Eq. (5.8)]. In 1 18], an IM/DD link 
analysis using 4-PAM signaling was performed. In |T4| , upper 
and lower bounds on the capacity of 2-, 4-, 8-, and 16-PAM 
were derived and in 1 19], the power efficiency of M-PAM was 
shown to be low. The M-ary pulse-position modulation (M- 
PPM) are known to be power-efficient; however, they have a 
poor spectral efficiency |11], |20|. 

Since any nonnegative electrical waveform satisfies the 
above channel constraint, it can be communicated successfully 
over an IM/DD link. This implies that if the information to 
be transmitted is firstly modulated on a subcarrier (electrical) 
using any M-level modulation format, it can be transmitted 
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Fig. 1: (a) Baseband transceiver with constrained-input Gaus- 
sian channel, (b) Passband transceiver of IM/DD systems. 



on an IM/DD Hnk after adding a direct current (DC) bias 
to ensure its nonnegativity, i.e., the subcarrier ampHtude and 
phase which carries the information can be retrieved at the 
receiver. This concept is known as subcarrier modulation 
(SCM) and was described in the wireless infrared commu- 
nication context ||T0| Ch. 5]. Therefore, the power efficiency 
compared to M-PAM can be improved since SCM allows 
the use of power-efficient multilevel modulation formats with 
IM/DD systems. In | [2T|, the SCM concept is experimentally 
demonstrated, and in f22], a novel transmitter design for the 
subcarrier quadrature phase- shift keying (QPSK) and 16-ary 
quadrature amplitude modulation (16-QAM) is presented. The 
DC bias required to ensure the nonnegativity of the electrical 
waveform does not carry information |TOl Ch. 5], | [2T| , p2| . 
Therefore, the improved power efficiency can be achieved by 
allowing the DC bias to vary on a symbol-by-symbol basis and 
within the symbol interval as in | 23 | and |24|, respectively. By 
guaranteeing nonnegativity, the investigation of lattice codes 
for IM/DD with AWGN became feasible and this is explored 
in |T5l, p5|. In |[T5), a signal space model for optical IM/DD 
channels is presented, where average and peak optical power 
are considered as design constraints for constructing lattice- 
based modulation formats. In addition, constellation shaping 
to reduce the average optical power has been studied in p5| 
for the case where no amplification is used, and in f26l where 
optical amplifiers are used. 

In this work, we optimize IM/DD modulation formats with 
and without confining them to a lattice structure. We propose 
a set of 4- and 8-level single- subcarrier modulation formats 
which are optimized for average electrical and optical power. 
These optimization criteria are both relevant, because the 
average electrical power is the standard power measure in 
digital and wireless communications |27, p. 40] and it helps 
in assessing the power consumption in optical communica- 
tions 1 28 1, while the average optical power is an important 
figure of merit for skin- and eye-safety measures in wireless 
optical links |10, Ch. 5], fTPl, fTSl and for quantifying the 
impact of shot noise in fiber-optical communications p. 
20]. 

II. System Model 

The system model under study is depicted in Fig. [TJ^)- It 
consists of a modulator which maps the data symbols u{k) 



at instant /c to a waveform belonging to the signaling set 
S = {5o(^), . . . , 5m-i(^)}, where M is the size of the 
signaling set. The generated waveform 



= sr[i]{t-lTs), 



(1) 



where r[/] is an ergodic process uniformly distributed over 
{0,1,...,M— 1} and is the symbol period, is constrained 
to being real and nonnegative. The received signal can be 
written as 

y{t)=x{t)^n{t), (2) 

where n{t) is a zero-mean Gaussian process with double- sided 
power spectral density No/2. It should be noted that there 
exists no nonnegativity constraint on the signal y{t). This 
is then followed by the demodulation of y{t) which yields 
u{k), Sin estimate of u{k). The demodulator is a correlator 
or matched filter receiver, which minimizes the symbol error 
rate at a given signal-to-noise ratio (SNR) ||^ Sect. 4.1]. This 
model is different from the conventional AWGN channel by 
the fact that the input x{t) is constrained to being nonnegative. 

The baseband model in Fig. [TJa) has been extensively 
studied in the optical communications context, since it serves 
as a good model for intensity-modulated direct-detection 
(IM/DD) systems jTO] Ch. 5], |^, |[Tg-||T3. The passband 
transceiver for IM/DD systems is depicted in Fig.[TJb). In such 
systems, the electrical nonnegative waveform x{t) directly 
modulates a light source, such as a laser diode. Therefore, the 
information is carried on the envelope of the passband signal 
z{t) = ^/2cx{t) cos(27r/ot+^), i.e., the intensity of the optical 
field, where c represents the electro-optical conversion factor 
in watts per ampere (W/A) |29, pp. 1-33], jIT] pp. 32- 
67], fo is the optical carrier frequency, and ^ is a random 
phase, uniformly distributed in [0,27r). It then propagates 
through the optical medium depicted as an optical fiber in 
Fig. [TJb), which could be a free- space optical link in other 
applications. At the receiver, the photodetector detects the 
power of z{t). Since the dominant channel impairment in 
optical IM/DD systems is the thermal noise resulting from the 
optical-to-electrical conversion |26| , p2l p. 155], the received 
electrical signal can be written as 



y[t) = rcx{t) + n(t), 



(3) 



where r is the responsivity of the opto-electrical converter 
in A/W. Without loss of generality, we set rc = 1, which 
yields ([2]). 

III. Signal Space Model 

By defining a set of orthonormal basis functions (/)/c(t) for 
k = 1, 2, . . . , and N < M sls in [ISj , each of the signals 
in S can be represented as 

N 

Si{t) = Ys,^kMt) (4) 

k=l 

for i 0, ...,M - 1, where = (5^,1, 5^,2, ^i, at) 
is the vector representation of Si{t) with respect to the 



aforementioned basis functions. Therefore, the constellation 
representing the signaling set S can be written as 1^ = 
{So,Si, . . . ,sm-i}. 

A. Single-Subcarrier Modulation Formats 

For in-phase and quadrature phase (I/Q) modulation formats 
to be used on IM/DD channels, a DC bias is required in 
order for x{t) to be nonnegative. This could be translated 
geometrically by having a three-dimensional (3d) Euclidean 
space spanned by the orthonormal basis functions 
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where 



rect(t) 



' 1 if < t < 1 
otherwise 



and / is the electrical subcarrier frequency fT5\. The basis 
function represents the DC bias, where Si^i is chosen 

for each z = 0, . . . , M — 1 such that 



min Si{t) > 0, 



which guarantees the nonnegativity of x{t) in ([T]). However, 
^2(^) and ^3(t) are the basis functions of the conventional 
I/Q modulation formats such as M-PSK and M-QAM. As 
in flO, pp. 115-116] and [15 1, we use / = l/T^, which is 
the minimum value for which (/)2(^), and ^3(t) are 

orthonormal. In |15|, IM/DD modulation formats based on 
these three basis functions are referred to as raised-QAM, and 
in p3| as single-cycle SCM. 



B. Performance Measures 

Two important power performance measures can be ex- 
tracted from the baseband and passband models in Fig. [T] The 
first entity is the average electrical power defined as 

1 r 

Pe = \im —- x^(t)dt, 

which for any basis functions can be simplified to 

^ E, 1 



E[||s.||^], 



(6) 



where Eg is the average energy of the constellation and E[-] 
is the expected value. This entity is an important figure of 
merit for assessing the performance of digital and wireless 
communication systems pT) p. 40]. Therefore, it is relevant 
for IM/DD systems for compatibility with classical methods 
and results p4| , | [35| . In addition, it helps in quantifying 
the impact of relative intensity noise (RIN) in fiber-optical 
links ^291, pp. 1-33], and in assessing the power consumption 



of optical systems |28| . In |36| , Pg was used as a performance 
measure for comparing different intensity modulation formats. 

The second measure is the average optical power P^, which 
has been studied in pO| , (TTj, |p4|-p6| for the wireless 
optical channel. Limitations are set on Pq for skin- and eye- 
safety standards to be met. In fiber-optic communications, this 
entity is used to quantify the impact of shot noise on the 
performance |29, p. 20]. It is defined as 
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This measure depends solely on the DC bias required to make 
the signals nonnegative and can be represented in terms of the 
symbol period and the constellation geometry as p4| , p5| 



Pn = 



(7) 



In order to have a fair comparison between the different 
modulation formats, the spectral efficiency defined as 7/ = 
Rh/W (bits/s/Hz) should be taken into account, where = 
Rg log2 M is the bit rate, Rg = 1 /Ts is the symbol rate, 
and W is the baseband bandwidth defined as the first null in 
the spectrum of x{t). The term baseband bandwidth is due 
to the fact that the baseband model in Fig. [TJa) hides the 
high frequency nature of the optical carrier |10, pp. 109-116]. 
At the same symbol rate, modulation formats such as OOK 
and M-PAM have W = R., whereas the modulation formats 



belonging to the single- subcarrier family in Sec. |III-A] occupies 
W = 2Rs\ this is due to the intermediate step of modulating 
the information onto an electrical subcarrier before modulating 
the optical carrier |10, Ch. 5], |36|. 

IV. Constellation Optimization 

For designing constellations, the admissible region defined 
in p5| Eq. (10)] as the set of all signals satisfying the 
nonnegativity constraint has to be taken into account. Given 
the 3d-Euclidean space defined for SCM in Sec. 
admissible region can be written as 



III-A the 



T = {s G : 4i > 2{sl^ 



si,)}. 



(8) 



which is a 3d-cone with apex angle of cos~^(l/3) = 70.528° 
pointing in the dimension spanned by (j)i(t). 

As done before for the conventional AWGN channel |(5|- 
(91, our approach of finding the best constellations can be 
formulated as a sphere-packing problem with the objective of 
minimizing a cost function depending on the constraints that 
might be present in the system model shown in Fig. [T] Thus, 
the optimization problem, for given constants M and d^^^, can 
be written as 



Minimize 
Subject to 



\n\ = M 

1^ C T 

cZ(f2) = cZmin, 



(9) 
(10) 
(11) 
(12) 



Fig. 2: (left to right): ^4 or ^4, ^p„8. and ^g- 



where 



(i(r^) = min | 



Choosing the objective function as ^(1^) = E[||si|p] results 
in ft = ^p^^M^ i-^-' a constellation optimized for average 
electrical power, and = IE[si,i] results in ^7 = ^p^^m^ 
i.e., a constellation optimized for average optical power. The 
constraint in ( pTj ) guarantees that the signals belong to the 
admissible region T, therefore satisfying the nonnegativity 
criterion of the channel. The minimum distance dmin serves 
as a good measure of error probability performance in the 
presence of AWGN at high SNR. Although this optimization 
problem is well formulated mathematically, it is rather difficult 
to obtain an analytical solution. Therefore, we resorted to 
numerical optimization techniques as in to find the 

best constellations. The optimization problem is nonconvex; 
therefore, a local solution does not imply that it is global. 

A special case of this optimization problem, which might 
not guarantee the optimal solution, is to confine the possible 
constellations to have a regular structure such as that of a 
lattice, denoted by A. In this case, the above optimization prob- 
lem can be reformulated by replacing ( pJ^ with Q c T D A, 
and dropping (12) since it is directly inferred by ^U) . By 
using the face-centered cubic lattice (A3), which provides the 
densest packing for the 3d-Euclidean space |3, p. xvi], we 
obtain ^p^^m^ and ^p^^m^ the constellations optimized for 
average electrical and optical power, respectively. 

A. Optimized Constellations 

Below is the description of the obtained constellations 
whose coordinates are included in App. [A| W e conjecture that 
all of them are optimal solutions of ([9|)-(|12[). 

1) A-lev el Constellations: Fig .J2] depicts the 4-level constel- 
lation which provides the lowest Pg and Pq while satisfying the 
optimization constraints. The geometry of this constellation is 
a regular tetrahedron where all the spheres, or the constellation 
points lying at the vertices of this regular tetrahedron, are 
equidistant from each other and normalized to unit dmin- This 
constellation is also the result of T fl A3. Since the obtained 
constellation is optimized for both Pg and P^, we will refer to 



it as "^4 or ^4. It is a remarkable fact that the vertex angle of 
the tetrahedron, defined as the apex angle of the circumscribed 
cone, is exactly cos~^(l/3), which is equal to the apex angle 
of the admissible region T. Thus, "^4 fits T snugly, in the sense 
that all constellation points, regarded as unit-diameter spheres, 
touch each other as well as the boundary of T, which, as we 
shall see in the next section, makes the modulation format 
very power-efficient. This modulation consists of a zero level 
signal and a biased ternary PSK. In |36|, a power-efficient 
modulation format called on-off phase- shift keying (OOPSK) 
was presented. It turns out that it has the same geometry as 
"^4. Other hybrids between amplitude- shift keying and PSK 
have been studied in (STj and | [38| ; however, such modulation 
formats do not satisfy the nonnegativity constraint of IM/DD 
channels. 

2) 8-level Constellations: Fig. [2] also shows the 8-level 
constellations ^p^^s and ^p^^s- None of these constellations 
are lattice-based, but all of them contain ^4 as the lowest 
four spheres. The highly symmetric and compact constellation 
^Pe,8 consists of four central spheres arranged in a tetrahe- 
dron and four additional spheres, each touching three spheres 
in the central tetrahedron. Surprisingly, seven of the eight 
spheres touch the conical boundary of T. This modulation 
is a hybrid between 2-PAM and two ternary PSK which are 
DC biased differently. The constellation ^p^^s the same as 
^P^^s hut with one sphere moved. On the other hand, when 
confining the set of points to a lattice structure, the resulting 
constellations which provide the lowest Pg and Po are the 
same, = '^p^,8 — ^Po,8^ and is depicted in Fig. [2] This 
constellation also contains ^4 as the lowest four spheres. 

B. Previously Known Constellations 

Our investigation encompasses other previously known for- 
mats which are presented after being normalized to unit dj^nn- 
At spectral efficiency r] = 1 bits/s/Hz, OOK defined as 
{(0), (1)} in terms of ^i(t) will be compared with and 
with subcarrier QPSK defined as {(1, ±1/2, ±1/2)} in terms 
of the basis functions defined in Sec. IIII-AI 

At a spectral efficiency r] = 3/2 bits/s/Hz, ^p^ g, ^p^ g, 
and ^8 will be compared with subcarrier 8-PSK defined 
as (l/sin(7r/8)){(l/V^,cos(7rz/4)/2,sin(7rz/4)/2)} for i = 
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Fig. 3: Simulated (solid) and theoretical (dotted) SER for the 
various modulation formats vs. 7^^ (top), jp^ (bottom). 



0, . . . , 7, star- shaped 8-QAM p7| with a constant bias defined 

as {((1 + V3)/V2,±l/2,±1/2),((1 + V3)/V2,0,±(l + 
^/3)/2), ((1 + V3)/V2,±{1 + V^)/2,0)}. We also include 
in our analysis a star-shaped 8-QAM denoted as 8-QAM 
in which the DC bias is allowed to vary from symbol 
to symbol, thus carrying information, and is defined as 
{(1, ±1/2, ±1/2), ((1 ± V^)/V2,0,±(l ± V^)/2),((l ± 
V3)/V2,±(l± V3)/2,0)}. 

V. Performance Analysis 

The symbol error rate (SER) performance of the different 
modulation schemes will be assessed. The standard union 
bound found in (27] Eq. (4.81)] is used to approximate the 
theoretical SER. This union bound can be approximated as 



(13) 



where K is the number of distinct signal pairs {si{t)^ Sj{t)) 
with i < j for which J{si{t) - Sj(t)Ydt = d^^^. This 




approximation is tight at high SNR. 

Fig. [3] (top) shows the simulated and theoretical SER of the 
studied modulation formats vs. SNR defined as 



7^, = 101ogio|^. 



(14) 



Apparently, all the modulation formats which are optimized 
for Pg outperforms the other formats at the same spectral 
efficiency. For spectral efficiency 7/ = 1, has a 0.86 dB 
average electrical power gain over OOK and 2.87 dB gain 
over QPSK to achieve = 10"^. For r] = 3/2, § has a 
0.3 dB gain over ^p^^g, 0.58 dB gain over 2.55 dB gain 
over 8-QAM, 4.35 dB gain over 8-QAM, and 4.39 dB gain 
over 8-PSK to achieve Pg = 10~^. The modulation formats 
optimized for Pg and Pq are very close in performance to 
the lattice-based modulation format. 

In order to facilitate the comparison of modulation formats 
in terms of their average optical power requirements, we define 
the optical SNR as 



7p^ = lOlogi 



Po 



(15) 



in a similar fashion as in Eq. (5)] . Fig. [3] (bottom) shows 
the SER plotted vs. jp^. Quite obviously, the modulation 
formats optimized for Pq perform better than the rest. For 
T] = I, ^4 has a 0.43 dB average optical power gain over 
OOK, and a 2.06 dB gain over QPSK to achieve an SER of 
10-^. For T] = 3/2, 8 has a 0.04 dB gain over ^p^ 
0.46 dB gain over ^g, 1.35 dB gain over 8-QAM, 2.48 dB 
gain over 8-PSK, and a 2.75 dB gain over 8-QAM to achieve 
a Ps = 10~^. Besides the very close performance of ^p^ § 
and ^p^^s^ ll is clear that they together with the lattice-based 
modulation perform better than the other modulation formats 
under study in terms of average optical power performance. 

VI. Conclusion 

By relaxing the constraint on the set of points used to 
design modulation formats for IM/DD channels, we were able 
to design 4- and 8-level modulation formats which are more 
power-efficient than known ones. For the 4-level modulation 
formats, the most power-efficient modulation in terms of 
average electrical and optical power happens to have a lattice 
structure even though the number of constellation points is 
small. This constellation is also a subset of all the obtained 
higher level constellations. As for the 8-level constellations, 
power-efficient schemes are obtained by not confining the set 
of constellation points to a lattice structure. However, this 
comes at the price of losing the geometric regularity found 
in the lattice structure. We conjecture that the new modulation 
formats are optimal for their size and optimization criteria over 
IM/DD channels. 

Appendix A 
Obtained Constellations 

Constellations are normalized to unit dmin- 



(72/3, 0, 1/^/3), ( V^, ±1/2, -a/3/6)}. 
%,8 =^4 U {((5/3)7273, 0, -5/(3^3)), 

((5/3)^273, ±5/6, 5/(6^3)), (2^2/3, 0, 0)}. 
^P^,8 =^4 U {((5/3)72/3, 0, -5/(3^3)), 

((5/3)72/3, ±5/6, 5/(6^)), 

(1.6293,0.9236,-0.6886)}. 
^P^,8 =^P^,8 = ^4 U {(27273, ±1/2, V3/6), 

(27273, 0, -1/73), (27273, 1, -1/V3)}. 
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